Hi @cdprete,

I created a new PR for issue #4342, as I realized my previous approach would not be sufficient to implement this feature cleanly.

I think the previous PR(#5326) can be closed. Could you please review the new PR and let me know if any further changes are needed?

It took longer than I initially expected because of the edge cases involved.

Hi @vetri15.

You can close the old PR yourself if you think the new one supersedes it.

Could you please review the new PR and let me know if any further changes are needed?

I try to do it asap.

It took longer than I initially expected because of the edge cases involved.

No worries.

@SteKoe @erikpetzold @ulischulte @hzpz can you maybe allow the pipeline to run in the meantime so that we make sure there are no potential issues? Tnx

Yeah, you were right @cdprete.

I have closed the old PR.

By the way, please be aware that there are issues with empty lines.
I don't know if your PR already covers it somehow, but for details have a look at #5388.

Hi @cdprete,

Thanks for pointing this out. I checked #5388 and updated this PR to handle empty log lines in the logfile view as well.

Empty strings are now rendered as <br> instead of an empty <pre>, so blank lines from the log file are preserved visually. I also brought over the related logfile styling from #5388 for tr, td, br, pre, and logfile links.

Hi @cdprete,

Thanks for pointing this out. I checked #5388 and updated this PR to handle empty log lines in the logfile view as well.

Empty strings are now rendered as <br> instead of an empty \<pre\>, so blank lines from the log file are preserved visually. I also brought over the related logfile styling from #5388 for tr, td, br, pre, and logfile links.

Hi @vetri15.

I just wanted to inform you I updated the styles in my PR as follows:

• removed w-full from pre and br since on tr and td is already enough
• changed the on-hover target from pre to td to respect also the empty new lines, but it seems like it's already working on your branch as it should

I tested this a bit and discovered the following bugs/issues:

• the first line in the log file is lost:

• with the tailing disabled, entering the manual navigation mode requires to press the down arrow twice but, for some reason, if then the up arrow is pressed ('cause I want to check the previous chunk for example), it's again required to press the down arrow twice to properly navigate the content
• the manual navigation puts the scrollbar at the bottom by default, while ideally should be at the top 'cause you want to read the current chunk top to bottom and not have to scroll with scrollbar to the top then to read it. Especially because it jumps over quite some lines
• no errors are shown anymore, while they're there in the console. While this may be wanted for some specific errors (e.g.: 416), even errors like "Connection Refused" are not shown anymore, therefore someone presses the arrows but nothing really happens. To simulate this, just stop the backend server.
• On server reboot it appears and the UI doesn't really recover from it unless it's manually reloaded, even if the server is up in the meantime:

• As soon as the manual navigation renders some remaining lines on, let's say, a new page (so not a continuation of the current one) the styles are broken. No scrollbars, no w-full and so on:

• I think there may be some issues on how the ranges are calculated. Even by using the previous follow/tail mode, there are now way more 416 in the console:

• The tooltip on the follow/tail button always says "Resume following" which is wrong if such a mode is already active

Hi @cdprete , I have summarized the issues you mentioned before. have corrected most of them. need your input on the remaining ones.

Hi @vetri15.

I was not able to replicate it , however my suspicion was when switching from follow mode to manual mode , the $nextTick(() => this.scrollToBottom();) from follow mode is spilling over to manual mode for one tick, so I added a guard check. I hope this is solved , if not please let me know

I'll give it a new try as soon as I've some more time and I let you know.

Need some help replicating this scenario , I am testing this on a random springboot where if i stop and start an application it will start with a new instance Id. after doing some research, even in cloud environments only some carry the meta data like instance Ids when restarting. need your input on how to tackle this.

I'm using the servlet sample which is already part of the project. Maybe it helps.

I am not able to understand clearly , in manual mode we are traversing either previous or next chunk so either way it would be a continuation. correct me if my understanding is wrong

Yes and no. I mean, if "remaining" is not 0, you're now fetching it and rendering it.
As soon as this is the case, as it can be seen in the screenshot above, instead of rendering it as a continuation the new lines get somehow rendered as a new page and the styles are broken.

I am not able to replicate it , however while it might be cause. I did not handle the logfile compression in the ManualMetadataPolling , I have corrected it so the 416 error spam should be gone.

I'll give it a new try as soon as I've some more time and I let you know.

Hi @vetri15.

This (it's the latest version you committed, btw) maybe shows better what I meant with the styles getting broken:

As you can see in the red section, w-full is somehow not applied anymore, but only as soon as the manual navigation is used to see a remaining chunk.
The same is true for the missing scrollbars which are, instead, present in follow mode.

Also, I can still see 416 responses flooding the console:

,the log file has NOT been compressed yet and, for what I can see, it's one error every 1s roughly.
See the counter of the failed requests going up in the video below:

The navigation seems to be worse now than in the previous version, in particular in the following flow:

• stop the follow mode and let it accumulate some logs
• press the arrow down to render the chunk which can lead to 3 scenarios:

1. the down arrow is still active and, pressing it again, produces the rendering of just other X lines without concatenating them to the previous ones, even if there is plenty of space in the page
2. the down arrow is disabled --> let it accumulte some more logs and repeat --> on click it will just render the new lines without appending them to the existing ones (the same as in 1)
3. the down arrow is disabled --> let it accumulte some more logs and repeat --> on click it will happen the new line (maybe because it's only one?) to the existing ones, therefore working as expected (but I'm not certain of this, I need to carefully check the log lines)

See below:

Also, unless I've maybe too few logs, the navigation between chunks is not really possible. On page up, the full logs gets then rendered again, instead of just the previous chunk which makes me wonder what happens if the log file has GB of text in it.

Hi @cdprete

my bad, I missed that style missing. I have corrected that in my local repo. will push it once all issues are solved. I am working on the UI auto recovering on stop and start of a server. have decided to do a retry logic for a few seconds before resetting the page. so there will be no need of manual refresh.

416 spam error:

Also, I can still see 416 responses flooding the console:

Thanks for the insight , I was able to pinpoint the cause of error. when appending new bytes if the request is on the edge of the byte range , it is causing the error. previously I was testing with a slightly higher frequency quartz job , so it went unnoticed.
Once I clear these bugs, I’ll let you know.

Regarding that, Manual file navigation rendering few lines , it was intentional based on the following discussion on the previous PR (#5326), #5326 (comment),

If you prefer filling the table approach , let me know. However, that would make the code a bit more complex.

Hi @cdprete ,

I have updated the PR and implemented fixes for the remaining bugs.

For navigation near the start or end of the file, I am overlapping bytes to keep the table filled using the existing 300 KB chunk size.

regarding the table not being filled near start and end: Rendering exactly the number of bytes needed to fill the table is difficult because wrapped lines take uneven vertical space. Zooming in or out also changes how many lines are needed to fill the visible table area. So keeping a fixed 300 KB chunk with boundary overlap is a more stable approach.

regarding the table not being filled near start and end: Rendering exactly the number of bytes needed to fill the table is difficult because wrapped lines take uneven vertical space. Zooming in or out also changes how many lines are needed to fill the visible table area. So keeping a fixed 300 KB chunk with boundary overlap is a more stable approach.

I know, that's why I said it would be ideal but I didn't request it.

3. the manual navigation puts the scrollbar at the bottom by default

I'll test it once more and make a video showing the issue if it still occurs.

Hi @vetri15.

I tested it again and I cannot reproduce anymore the issue with the scrollbar going at the bottom while using the pagination.

Implemented a retry mechanism , that keeps retrying until network is back again , if it is back withing 10 retries , it would continue with normal followmode , if it comes online after more than 10 retries it will do a complete reset , kind of like a page refresh and starts following logfile.

Why such a distinction? Are you sure it's really the case?
I stopped the server for quite some time and, after restarting it, the manual navigation mode was still active (which is correct in my opinion).

Moreover, what's the interval between retries? Ideally it shouldn't be too short (e.g.: 1s or more, but not less).
Nevertheless, such a reconnection also works, at least when restarting the application from the IDE.
I only think, though, that after a certain max value, we should just stop retrying and show an error.

I would keep retrying forever only if the error is due to SBA server itself being down.
In that case the overhead is fully and only on the browser so I wouldn't mind.
But, if it's the app that SBA calls to be down, then SBA server would have to deal with quite an overhead that it should be avoided IMO.

@SteKoe, maybe you can provide more insights, but I wonder if we get different responses in the UI on the specific case of connection failure depending on the fact that the failure comes from SBA server itself or from the proxied/monitored application.
Ideally the response status code should be different, no? If it's the case, we could indeed differentiate between the 2 cases and handle them differently, by being on the safe side if it's the monitored app to be down to not overload SBA server.

Only one note: while it was trying to reconnect, I got from a short moment an error mentioning 503 instead of just Network error and it seems like it gets shown as soon as the connection gets established again.
I can't say why, as of now.

Moreover, while the reconnection logic is in progress, all the buttons apart the line wrap should be disabled.
Otherwise, they can be pressed and then the state simply breaks.

Also I just realized that now I've the opposite problem :D
Given that the scrollbar stays on top on page down, I've to manually scroll to the bottom by hand to just see the last added line :D

Mmm I cannot really imagine a solution for this... unless to implement something driven on the number of remaining bytes on a best effort. Like, if it's more or less one full "page" then keep it to the top (as it is now, so that by scrolling manually you can read the content of the "page"), but if it's just some lines then scroll to the bottom.
Dunno how feaseable it is, though.

Hi @cdprete,

Thanks for pointing out the button disabling issue , I have now disabled it. and updated the PR.

Moreover, while the reconnection logic is in progress, all the buttons apart the line wrap should be disabled.
Otherwise, they can be pressed and then the state simply breaks.

Why such a distinction? Are you sure it's really the case?
I stopped the server for quite some time and, after restarting it, the manual navigation mode was still active (which is correct in my opinion).

You are right, manual navigation can stay active after a server restart. Manual mode only fetches explicit ranges and updates the total size metadata, so I left it to recover without a full reset.
In Follow mode, state such as the current byte range and remainders can get out of sync with the actual file, so it resets after retrying for around 10 * 1(default poll timer) = 10 seconds.

It seems like the 503 gets shown as soon as the connection gets established again.
I can't say why, as of now.

Yes, a short-lived 503 can happen during reconnect. The logfile request goes through SBA’s /instances/{id}/actuator/logfile proxy; if the instance is temporarily unavailable or not yet registered during restart, the proxy returns 503 Service Unavailable. Once the next retry succeeds, the error is cleared.

I just realized that now I've the opposite problem :D
Given that the scrollbar stays on top on page down, I've to manually scroll to the bottom by hand to just see the last added line :D

Yeah, looks like I fixed one side and created the mirror version of the problem :D

like you said , the ideal behavior would be to scroll to the bottom only when page down adds just a few lines. But tracking that accurately may be tricky, since the number of visible lines depends on screen size, wrapping, and font rendering.

I think we can skip that for now, unless you really think this should be handled in this PR.

Hi @vetri15

I think we can skip that for now, unless you really think this should be handled in this PR.

Sure, but then let's go to the previous behaviour that the scrollbar goes to the bottom.
At least it's consistent with the currently released behaviour and it will create less unexpected surprises or complains.
Also, the odds that very high log levels are enabled that creates many lines are small, so the existing behaviour to keep scrolling to the bottom is probably better.

Hi @cdprete ,

The manual navigation now scrolls to the bottom on navigation.

Hi @cdprete ,

The manual navigation now scrolls to the bottom on navigation.

Hi @vetri15.
Great, thanks.

Can you please align with the master branch by resolving the conflicts so that I'll do a final test and then let the team handle this?

The manual navigation now scrolls to the bottom on navigation.

Would it be possible, eventually, during the manual navigation, to first scroll to the top/botton and, only when the button is pressed again, fetch the previous/next chunk?
This arises, though, the careful enabling/disabling of the buttons.
For example, when we're rendering the first chunk but we're at the bottom (or, maybe easier, not at the top) of the page the up button should be active one last time so that, if pressed, scrolls to the top.
Similarly for the down button that, if we're rendering the last (so far) chunk but we're not at the top of the page, it should be enabled one last time.

I think that in the previous PR I said I was against this idea, but from a UX prospective I find it way better than having to manually scroll on every page. Also, it's more aligned with the previous behaviour.

Hi @cdprete,

Would it be possible, eventually, during the manual navigation, to first scroll to the top/botton and, only when the button is pressed again, fetch the previous/next chunk?

Yes, definitely. It should be possible to first scroll to the top/bottom during manual navigation, and only fetch the previous/next chunk when the button is pressed again.

I’ll work on this and update the PR.

Clarification needed: since we have set scroll to bottom as default on manual navigation , when we traverse using page up , we would need to press two times for traversing up. but when traversing page down , we would only press one time for each traverse down as the loaded page will already be at bottom. there will not be symmetry between traversing up and down, is it fine ?

I saw your comment only after merging master, so please ignore the previous merge.

Hi @vetri15,

but when traversing page down , we would only press one time for each traverse down as the loaded page will already be at bottom

I don't follow. The 1st page down would move the scrollbar to the bottom and the 2nd would fetch the next chunk, if available.
Am I missing something?

I saw your comment only after merging master, so please ignore the previous merge.

???

Hi @cdprete ,

The 1st page down would move the scrollbar to the bottom and the 2nd would fetch the next chunk, if available.
Am I missing something?

Yea , you are right. If the scroll bar is at the middle, whatever direction we click it will scroll to , and only on the second click, it will load the next/previous chunk.

My doubt was when we are traversing a big file and we are on one of the middle chunks with scroll bar also in middle,

1. If we decide to go up (previous), first click will scroll up , second click will load chunk, it will be scrolled down on the newly loaded chunk, so we need to press twice again to load the previous chunk.

2. If we decide to go down(next) , first click will scroll down , and the second click will load chunk , it will be scrolled down on the newly loaded chunk , so don't need the two click routine for traversing a single chunk.

I was talking about this asymmetry ,for consequent traversal : one click for down but two click for traversing up. I cant think of a simple solution to this to behave symmetrically

Hope it made sense

Hi @vetri15,

If we decide to go up (previous), first click will scroll up , second click will load chunk, it will be scrolled down on the newly loaded chunk, so we need to press twice again to load the previous chunk.

I would say: first click scrolls up, second click loads the chunk and scrolls up, so that the view makes visible the chunk from the beginning

If we decide to go down(next) , first click will scroll down , and the second click will load chunk , it will be scrolled down on the newly loaded chunk , so don't need the two click routine for traversing a single chunk.

I would say: first click scrolls down, second click loads the chunk and scrolls up, so that the view makes visible the chunk from the beginning

In a nutshell:

• 1st click: scroll in the direction of the arrow
• 2nd click: load the chunk and scroll up to make it readable from the beginning

I'm not sure about the scroll up for visibility purposes, though. I would need to see the outcome of it.

I was talking about this asymmetry ,for consequent traversal : one click for down but two click for traversing up.

This would be true only if you're already at the bottom and it's not really asymmetrical. It's just that when you're at the top you can't go further up.
Imagine that you could, it would require only one click as well.

Hi @cdprete ,

In a nutshell:
1st click: scroll in the direction of the arrow
2nd click: load the chunk and scroll up to make it readable from the beginning

Implemented the requested change , can you please check and tell whether it is working as expected

Hi @vetri15,

can you please check and tell whether it is working as expected

I'll give it a try tomorrow at max.

I've a question about the remaining bytes counter, though: is it there purely for display purposes or do you use it also for something?

I was thinking that, during the manual navigation, it may be an overkill.
Imagine I'm reading the chunk 1 and, in the meantime, other 10 chunks get written. It's quite wasteful to load them just to show a counter and, therefore, I would rather drop it.

On the other end, to make the manual navigation smoother, I would embrace the idea of prefetching and storing and previous and next chunk of the currently rendered one, so that on page up/down there are no delays. Of course, capped by a new line, so there may be the need to prefetch more than just one, or none at all if not available (e.g: we are on the 1st chunk -> no prev, we are on the last church and the pull of the next fails -> no next).
Probably the prefetching of the next chunk is trickier and requires careful implementation, though.

What do you think?

Hi @cdprete ,

I've a question about the remaining bytes counter, though: is it there purely for display purposes or do you use it also for something?

The remaining bytes counter is mostly display, but the underlying total size is also used to know whether the user can navigate to the next chunk. In manual mode we’re not fetching logfile contents for that. It just uses the existing poll timer to check the current logfile size/metadata. So it doesn’t load the chunks in between just to show the remaining bytes.

On the other end, to make the manual navigation smoother, I would embrace the idea of prefetching and storing and previous and next chunk of the currently rendered one, so that on page up/down there are no delays

I definitely like the adjacent prefetching idea, But I’d prefer to handle that as a separate issue/feature request, since it needs a bit more careful handling around partial lines, resets/compression, logfile growth, and failed prefetches. This PR is already getting a little complex/bigger than expected.

Let me know what you think.

Hi @vetri15,

In manual mode we’re not fetching logfile contents for that. It just uses the existing poll timer to check the current logfile size/metadata.

I don't follow. To check such values you have no other way apart calling /logfile with a range. But this will return that portion of the logs as well, even if it's then discarded.
I wonder if we could, at least, stop polling as soon as we know there is at least another chunk available.
Otherwise, depending on the log file content, we may download MB of data over the wire for nothing.
This is especially true when we hit a size which is bigger than the chunk size.

But I’d prefer to handle that as a separate issue/feature request, since it needs a bit more careful handling around partial lines, resets/compression, logfile growth, and failed prefetches.

Indeed. I was not asking to implement it as of now.

Hi @cdprete,

To check such values you have no other way apart calling /logfile with a range. But this will return that portion of the logs as well, even if it's then discarded

Not really , there is a shortcut :D , fetching range 0-0

HTTP/1.1 206 Partial Content
Content-Range: bytes 0-0/4707
Content-Length: 1

so the app just polls this endpoint , so does not download data , it just pings with the same frequency of polltimer config property used in follow mode , except this one does not fetch any content.

Hi @cdprete,

To check such values you have no other way apart calling /logfile with a range. But this will return that portion of the logs as well, even if it's then discarded

Not really , there is a shortcut :D , fetching range 0-0

HTTP/1.1 206 Partial Content
Content-Range: bytes 0-0/4707
Content-Length: 1

so the app just polls this endpoint , so does not download data , it just pings with the same frequency of polltimer config property used in follow mode , except this one does not fetch any content.

Hi @vetri15.

That's interesting.
I would have expected such a range configuration to fail.
How did you figure it out?
There is no mention of such an edge case in https://docs.spring.io/spring-boot/api/rest/actuator/logfile.html.

Hi @cdprete ,
Honestly, I didn’t find it. I asked ChatGPT how to find the size of the logfile. it gave me this trick and I checked , it worked :D .

That's interesting.
I would have expected such a range configuration to fail.

Yea , i was also skeptical too. but suppose if it failed i would have pestered the endpoint with 0-1 range , surely it can't reject that ;)

Hi @vetri15.

I tested it again and I realized that my idea to scroll to the top on the first page down is actually a bad idea, because, at least when using just the arrows, we're already at the bottom.
Or, to be precise, the idea would be good if we would be rendering the new chunk directly but, it seems like, that we render some old lines instead.
For example, the last line has time 12:12:41:ms but then, when the scrollbar gets moved to the top, I ended up with a line having 12:12:31:ms as time.
I was actually, instead, especting to get the next chunk rendered from the top and, therefore, the first line should have a time > 12:12:41:ms (with potentially only greater ms depending on the volume of the logs).

For the page up, this problem doesn't occur, though. But that's to be expected because, on page up, we are basically navigating through some history and therefore the data must be in the past.

I don't know how we could fix this. Any suggestions?

For the rest, it looks fine to me.

Hi @cdprete ,

For example, the last line has time 12:12:41:ms but then, when the scrollbar gets moved to the top, I ended up with a line having 12:12:31:ms as time.

Maybe this idea might work:

logic: scrolling dynamically based on direction, page down -> scroll bottom and for page up -> scroll to top.

page down:
case 1: trying to page down when near the file end. we press page down. it would load few bytes and there will be overlap since we are near file end. but since we scroll to bottom. the user won't get confused. suppose if they want to read from the top of page. they can just press up once and go to top.

case 2: trying to page down near the middle chunks. we press down. it would load next page and scroll to bottom. and next press would also do the same.

page up: remains unaffected.

this also inadvertently solves the asymmetry while traversing page i was talking about. since while going in one direction would automatically scroll in that direction , we would only need one click for one chunk on a big file.

what do you think about this approach ?

Hi @vetri15.

case 1: trying to page down when near the file end. we press page down. it would load few bytes and there will be overlap since we are near file end. but since we scroll to bottom. the user won't get confused. suppose if they want to read from the top of page. they can just press up once and go to top.

As soon as the overlap doesn't produce duplicated log entries in the view, it sounds ok.

case 2: trying to page down near the middle chunks. we press down. it would load next page and scroll to bottom. and next press would also do the same.

Won't this skip a chunk, though, because on the second click we scroll to the button and therefore jump at the end of the chunk, therefore reintroducing the previous issue that you've to scoll manually all the way up? Or am I missing something?

I kind of had the impression, but I've to try again, that the issue was occurring only when the remaining bytes were less (or equal, maybe) than the chunk size. If there were more remaining bytes than the chunk size then the scolling up was indeed positioning the log view at the first line of the new chunk.

what do you think about this approach ?

It it works I'm good with it.
Maybe you can give it a try, let me test it and I report back to you if the issue is solved or not.